Check insert privileges to create chunk #2416

mkindahl · 2020-09-18T11:38:04Z

To create a chunk in a hypertable, it is currently necessary to be the
owner of the hypertable of the chunk. If a user has insert privileges
only, it will fail with an error message, which causes problems when
inserting data into distributed hypertables since the user cannot
create new chunks.

This commit changes this and only requires that the user has insert
privileges on the hypertable of the chunk for allowing creation of a
new chunk.

Closes #2393

mkindahl · 2020-09-18T11:42:49Z

This cannot be reproduced in the tests since it requires construction of separate data nodes. It was reproduced manually with the following results:

Before commit:

postgres=# SET ROLE somebody;
SET
postgres=> INSERT INTO conditions
postgres-> SELECT time, 1+(random()*30)::int, random()*80
postgres-> FROM generate_series('2019-01-01 00:00:00'::timestamptz, '2019-02-01 00:00:00', '1 min') AS time;
ERROR:  [data_node_1]: must be owner of hypertable "conditions"

After commit:

postgres=# SET ROLE somebody;
SET
postgres=> INSERT INTO conditions
SELECT time, (random()*30)::int+1, random()*80
FROM generate_series('2019-01-01 00:00:00'::timestamptz, '2019-02-01 00:00:00', '1 min') AS time;
INSERT 0 44641

erimatnor

This cannot be reproduced in the tests since it requires construction of separate data nodes.

AFAIK, it should be possible to trigger this issue if you are a user, which is not the owner of the hypertable, and then insert a data point that creates a chunk on a data node. This shouldn't require separate database instances to reproduce.

It should also be possible to test the function and permissions in a limited way by simply calling _timescaledb_internal.create_chunk() directly in a test.

tsl/src/chunk_api.c

tsl/test/expected/chunk_api-12.out

codecov · 2020-09-18T12:03:37Z

Codecov Report

Merging #2416 into master will increase coverage by 0.14%.
The diff coverage is 95.40%.

@@            Coverage Diff             @@
##           master    #2416      +/-   ##
==========================================
+ Coverage   89.95%   90.10%   +0.14%     
==========================================
  Files         213      213              
  Lines       34354    34328      -26     
==========================================
+ Hits        30904    30931      +27     
+ Misses       3450     3397      -53

Impacted Files	Coverage Δ
src/catalog.h	`100.00% <ø> (ø)`
src/chunk_index.h	`100.00% <ø> (ø)`
src/dimension.c	`95.41% <ø> (ø)`
src/utils.c	`80.05% <ø> (ø)`
src/utils.h	`44.44% <ø> (ø)`
src/time_utils.c	`98.12% <88.88%> (+0.04%)`	⬆️
tsl/src/bgw_policy/continuous_aggregate_api.c	`92.19% <96.42%> (+0.28%)`	⬆️
src/bgw/job.c	`95.07% <96.96%> (+0.28%)`	⬆️
src/chunk_index.c	`94.72% <100.00%> (-0.03%)`	⬇️
tsl/src/chunk_api.c	`94.54% <100.00%> (+0.01%)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c15d8be...f65615b. Read the comment docs.

mkindahl · 2020-09-18T12:54:16Z

This cannot be reproduced in the tests since it requires construction of separate data nodes.

AFAIK, it should be possible to trigger this issue if you are a user, which is not the owner of the hypertable, and then insert a data point that creates a chunk on a data node. This shouldn't require separate database instances to reproduce.

It should also be possible to test the function and permissions in a limited way by simply calling _timescaledb_internal.create_chunk() directly in a test.

Managed to create one, but did it as a new test.

k-rus

An existing test was modified in a way that it doesn't make sense and confusing now.

tsl/test/expected/chunk_api-11.out

svenklemm

Please use an existing test suite like dist_hypertable

mkindahl · 2020-09-18T14:49:25Z

Please use an existing test suite like dist_hypertable

Merged it with dist_grants.

tsl/test/expected/chunk_api-12.out

tsl/test/expected/chunk_api-11.out

WireBaron

Looks good overall, but I'm curious who the owner of the newly created chunk is? It seems there might be issue if it's the inserting user.

tsl/test/expected/chunk_api-11.out

mkindahl · 2020-09-21T06:42:56Z

Looks good overall, but I'm curious who the owner of the newly created chunk is? It seems there might be issue if it's the inserting user.

This calls ts_create_chunk_table which says:

Create a chunk's table.

A chunk inherits from the main hypertable and will have the same owner. Since
chunks can be created either in the TimescaleDB internal schema or in a
user-specified schema, some care has to be taken to use the right
permissions, depending on the case:

To create a chunk in a hypertable, it is currently necessary to be the owner of the hypertable of the chunk. If a user has insert privileges only, it will fail with an error message, which causes problems when inserting data into distributed hypertables since the user cannot create new chunks. This commit changes this and only requires that the user has insert privileges on the hypertable of the chunk for allowing creation of a new chunk. Closes timescale#2393

This release adds major new features and bugfixes since the 1.7.4 release. We deem it moderate priority for upgrading. This release adds the long-awaited support for distributed hypertables to TimescaleDB. This release also adds support for user-defined actions allowing users to define actions that are run by the TimescaleDB automation framework. In addition to these major new features, the 2.0 branch introduces _breaking_ changes to APIs and existing features, such as continuous aggregates. These changes are not backwards compatible and might require changes to clients and/or scripts that rely on the previous APIs. Please review our updated documentation and do proper testing to ensure compatibility with your existing applications. The noticeable breaking changes in APIs are: - Function rename and arguments redesign for compression and retention policies - Creating continuous aggregate is split into 2 steps: `CREATE MATERIALIZED VIEW` to create a continuous aggregate view, and `add_continuous_aggregate_policy` to create the actual policy - Rename and argument redesign for functions to manipulate running policies - Replacing several information views with policy information with general views, which covers policies and user-defined actions - Redesign of information views This release candidate is upgradable, so if you are on a previous release (e.g., 1.7.4) you can upgrade to the release candidate and later expect to be able to upgrade to the final 2.0 release. However, please carefully consider your compatibility requirements _before_ upgrading. **Major Features** * timescale#1923 Add support for distributed hypertables * timescale#2006 Add support for user-defined actions * timescale#2435 Move enterprise features to community * timescale#2437 Update Timescale License **Minor Features** * timescale#2011 Constify TIMESTAMPTZ OP INTERVAL in constraints * timescale#2105 Support moving compressed chunks **Bugfixes** * timescale#1843 Improve handling of "dropped" chunks * timescale#1886 Change ChunkAppend leader to use worker subplan * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2263 Fix timestamp overflow in time_bucket optimization * timescale#2270 Fix handling of non-reference counted TupleDescs in gapfill * timescale#2325 Fix rename constraint/rename index * timescale#2370 Fix detection of hypertables in subqueries * timescale#2376 Fix caggs width expression handling on int based hypertables * timescale#2416 Check insert privileges to create chunk * timescale#2428 Allow owner change of continuous aggregate * timescale#2436 Propagate grants in continuous aggregates

This release adds major new features and bugfixes since the 1.7.4 release. We deem it moderate priority for upgrading. This release adds the long-awaited support for distributed hypertables to TimescaleDB. With 2.0, users can run multiple instances of TimescaleDB, configured so that one serves as an access node and multiple others as data nodes, so that all queries for a distributed hypertable are issued to the access node, but inserted data and queries are pushed down across data nodes for greater scale and performance. In addition to these major new features, the 2.0 branch introduces _breaking_ changes to APIs and existing features, such as continuous aggregates. These changes are not backwards compatible and might require changes to clients and/or scripts that rely on the previous APIs. Please review our updated documentation and do proper testing to ensure compatibility with your existing applications. The noticeable breaking changes in APIs are: - Redefined functions for policies - A continuous aggregate is now created with `CREATE MATERIALIZED VIEW` instead of `CREATE VIEW` and automated refreshing requires adding a policy via `add_continuous_aggregate_policy` - Redesign of informational views, including new (and more general) views for information about policies and user-defined actions This release candidate is upgradable, so if you are on a previous release (e.g., 1.7.4) you can upgrade to the release candidate and later expect to be able to upgrade to the final 2.0 release. However, please carefully consider your compatibility requirements _before_ upgrading. **Major Features** * timescale#1923 Add support for distributed hypertables * timescale#2006 Add support for user-defined actions * timescale#2435 Move enterprise features to community * timescale#2437 Update Timescale License **Minor Features** * timescale#2011 Constify TIMESTAMPTZ OP INTERVAL in constraints * timescale#2105 Support moving compressed chunks **Bugfixes** * timescale#1843 Improve handling of "dropped" chunks * timescale#1886 Change ChunkAppend leader to use worker subplan * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2263 Fix timestamp overflow in time_bucket optimization * timescale#2270 Fix handling of non-reference counted TupleDescs in gapfill * timescale#2325 Fix rename constraint/rename index * timescale#2370 Fix detection of hypertables in subqueries * timescale#2376 Fix caggs width expression handling on int based hypertables * timescale#2416 Check insert privileges to create chunk * timescale#2428 Allow owner change of continuous aggregate * timescale#2436 Propagate grants in continuous aggregates

This release adds major new features and bugfixes since the 1.7.4 release. We deem it moderate priority for upgrading. This release adds the long-awaited support for distributed hypertables to TimescaleDB. With 2.0, users can run multiple instances of TimescaleDB, configured so that one serves as an access node and multiple others as data nodes, so that all queries for a distributed hypertable are issued to the access node, but inserted data and queries are pushed down across data nodes for greater scale and performance. This release also adds support for user-defined actions allowing users to define actions that are run by the TimescaleDB automation framework. In addition to these major new features, the 2.0 branch introduces _breaking_ changes to APIs and existing features, such as continuous aggregates. These changes are not backwards compatible and might require changes to clients and/or scripts that rely on the previous APIs. Please review our updated documentation and do proper testing to ensure compatibility with your existing applications. The noticeable breaking changes in APIs are: - Redefined functions for policies - A continuous aggregate is now created with `CREATE MATERIALIZED VIEW` instead of `CREATE VIEW` and automated refreshing requires adding a policy via `add_continuous_aggregate_policy` - Redesign of informational views, including new (and more general) views for information about policies and user-defined actions This release candidate is upgradable, so if you are on a previous release (e.g., 1.7.4) you can upgrade to the release candidate and later expect to be able to upgrade to the final 2.0 release. However, please carefully consider your compatibility requirements _before_ upgrading. **Major Features** * timescale#1923 Add support for distributed hypertables * timescale#2006 Add support for user-defined actions * timescale#2435 Move enterprise features to community * timescale#2437 Update Timescale License **Minor Features** * timescale#2011 Constify TIMESTAMPTZ OP INTERVAL in constraints * timescale#2105 Support moving compressed chunks **Bugfixes** * timescale#1843 Improve handling of "dropped" chunks * timescale#1886 Change ChunkAppend leader to use worker subplan * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2263 Fix timestamp overflow in time_bucket optimization * timescale#2270 Fix handling of non-reference counted TupleDescs in gapfill * timescale#2325 Fix rename constraint/rename index * timescale#2370 Fix detection of hypertables in subqueries * timescale#2376 Fix caggs width expression handling on int based hypertables * timescale#2416 Check insert privileges to create chunk * timescale#2428 Allow owner change of continuous aggregate * timescale#2436 Propagate grants in continuous aggregates

This release adds major new features and bugfixes since the 1.7.4 release. We deem it moderate priority for upgrading. This release adds the long-awaited support for distributed hypertables to TimescaleDB. With 2.0, users can create distributed hypertables across multiple instances of TimescaleDB, configured so that one instance serves as an access node and multiple others as data nodes. All queries for a distributed hypertable are issued to the access node, but inserted data and queries are pushed down across data nodes for greater scale and performance. This release also adds support for user-defined actions allowing users to define actions that are run by the TimescaleDB automation framework. In addition to these major new features, the 2.0 branch introduces _breaking_ changes to APIs and existing features, such as continuous aggregates. These changes are not backwards compatible and might require changes to clients and/or scripts that rely on the previous APIs. Please review our updated documentation and do proper testing to ensure compatibility with your existing applications. The noticeable breaking changes in APIs are: - Redefined functions for policies - A continuous aggregate is now created with `CREATE MATERIALIZED VIEW` instead of `CREATE VIEW` and automated refreshing requires adding a policy via `add_continuous_aggregate_policy` - Redesign of informational views, including new (and more general) views for information about policies and user-defined actions This release candidate is upgradable, so if you are on a previous release (e.g., 1.7.4) you can upgrade to the release candidate and later expect to be able to upgrade to the final 2.0 release. However, please carefully consider your compatibility requirements _before_ upgrading. **Major Features** * timescale#1923 Add support for distributed hypertables * timescale#2006 Add support for user-defined actions * timescale#2435 Move enterprise features to community * timescale#2437 Update Timescale License **Minor Features** * timescale#2011 Constify TIMESTAMPTZ OP INTERVAL in constraints * timescale#2105 Support moving compressed chunks **Bugfixes** * timescale#1843 Improve handling of "dropped" chunks * timescale#1886 Change ChunkAppend leader to use worker subplan * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2263 Fix timestamp overflow in time_bucket optimization * timescale#2270 Fix handling of non-reference counted TupleDescs in gapfill * timescale#2325 Fix rename constraint/rename index * timescale#2370 Fix detection of hypertables in subqueries * timescale#2376 Fix caggs width expression handling on int based hypertables * timescale#2416 Check insert privileges to create chunk * timescale#2428 Allow owner change of continuous aggregate * timescale#2436 Propagate grants in continuous aggregates

This release adds major new features and bugfixes since the 1.7.4 release. We deem it moderate priority for upgrading. This release adds the long-awaited support for distributed hypertables to TimescaleDB. With 2.0, users can create distributed hypertables across multiple instances of TimescaleDB, configured so that one instance serves as an access node and multiple others as data nodes. All queries for a distributed hypertable are issued to the access node, but inserted data and queries are pushed down across data nodes for greater scale and performance. This release also adds support for user-defined actions allowing users to define actions that are run by the TimescaleDB automation framework. In addition to these major new features, the 2.0 branch introduces _breaking_ changes to APIs and existing features, such as continuous aggregates. These changes are not backwards compatible and might require changes to clients and/or scripts that rely on the previous APIs. Please review our updated documentation and do proper testing to ensure compatibility with your existing applications. The noticeable breaking changes in APIs are: - Redefined functions for policies - A continuous aggregate is now created with `CREATE MATERIALIZED VIEW` instead of `CREATE VIEW` and automated refreshing requires adding a policy via `add_continuous_aggregate_policy` - Redesign of informational views, including new (and more general) views for information about policies and user-defined actions This release candidate is upgradable, so if you are on a previous release (e.g., 1.7.4) you can upgrade to the release candidate and later expect to be able to upgrade to the final 2.0 release. However, please carefully consider your compatibility requirements _before_ upgrading. **Major Features** * #1923 Add support for distributed hypertables * #2006 Add support for user-defined actions * #2435 Move enterprise features to community * #2437 Update Timescale License **Minor Features** * #2011 Constify TIMESTAMPTZ OP INTERVAL in constraints * #2105 Support moving compressed chunks **Bugfixes** * #1843 Improve handling of "dropped" chunks * #1886 Change ChunkAppend leader to use worker subplan * #2116 Propagate privileges from hypertables to chunks * #2263 Fix timestamp overflow in time_bucket optimization * #2270 Fix handling of non-reference counted TupleDescs in gapfill * #2325 Fix rename constraint/rename index * #2370 Fix detection of hypertables in subqueries * #2376 Fix caggs width expression handling on int based hypertables * #2416 Check insert privileges to create chunk * #2428 Allow owner change of continuous aggregate * #2436 Propagate grants in continuous aggregates

mkindahl requested a review from erimatnor September 18, 2020 11:38

mkindahl force-pushed the dist_writer_perm branch from 6c7301c to f9120be Compare September 18, 2020 11:49

erimatnor reviewed Sep 18, 2020

View reviewed changes

tsl/src/chunk_api.c Outdated Show resolved Hide resolved

tsl/test/expected/chunk_api-12.out Show resolved Hide resolved

mkindahl force-pushed the dist_writer_perm branch from f9120be to cec21ad Compare September 18, 2020 12:53

mkindahl requested a review from erimatnor September 18, 2020 12:54

mkindahl self-assigned this Sep 18, 2020

mkindahl marked this pull request as ready for review September 18, 2020 12:54

mkindahl requested a review from a team as a code owner September 18, 2020 12:54

mkindahl requested review from k-rus and WireBaron and removed request for a team September 18, 2020 12:54

k-rus suggested changes Sep 18, 2020

View reviewed changes

tsl/test/expected/chunk_api-11.out Outdated Show resolved Hide resolved

svenklemm requested changes Sep 18, 2020

View reviewed changes

mkindahl requested review from k-rus and svenklemm September 18, 2020 14:49

k-rus approved these changes Sep 18, 2020

View reviewed changes

svenklemm reviewed Sep 18, 2020

View reviewed changes

tsl/test/expected/chunk_api-12.out Outdated Show resolved Hide resolved

svenklemm reviewed Sep 18, 2020

View reviewed changes

tsl/test/expected/chunk_api-11.out Show resolved Hide resolved

WireBaron reviewed Sep 18, 2020

View reviewed changes

tsl/test/expected/chunk_api-11.out Outdated Show resolved Hide resolved

mkindahl requested a review from svenklemm September 21, 2020 06:35

mkindahl requested a review from WireBaron September 21, 2020 06:43

svenklemm approved these changes Sep 21, 2020

View reviewed changes

mkindahl force-pushed the dist_writer_perm branch from baab93d to 38119bd Compare September 21, 2020 12:56

mkindahl force-pushed the dist_writer_perm branch from 38119bd to f65615b Compare September 21, 2020 13:12

WireBaron approved these changes Sep 21, 2020

View reviewed changes

erimatnor approved these changes Sep 21, 2020

View reviewed changes

mkindahl merged commit c321fe0 into timescale:master Sep 21, 2020

mkindahl deleted the dist_writer_perm branch September 21, 2020 16:16

svenklemm mentioned this pull request Oct 5, 2020

Release 2.0.0-rc1 #2478

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check insert privileges to create chunk #2416

Check insert privileges to create chunk #2416

mkindahl commented Sep 18, 2020

mkindahl commented Sep 18, 2020

erimatnor left a comment

codecov bot commented Sep 18, 2020 •

edited

mkindahl commented Sep 18, 2020

k-rus left a comment

svenklemm left a comment

mkindahl commented Sep 18, 2020

WireBaron left a comment

mkindahl commented Sep 21, 2020

Check insert privileges to create chunk #2416

Check insert privileges to create chunk #2416

Conversation

mkindahl commented Sep 18, 2020

mkindahl commented Sep 18, 2020

erimatnor left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 18, 2020 • edited

Codecov Report

mkindahl commented Sep 18, 2020

k-rus left a comment

Choose a reason for hiding this comment

svenklemm left a comment

Choose a reason for hiding this comment

mkindahl commented Sep 18, 2020

WireBaron left a comment

Choose a reason for hiding this comment

mkindahl commented Sep 21, 2020

codecov bot commented Sep 18, 2020 •

edited